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ABSTRACT 

This brief describes new requirements for vocational 
education program improvement and evaluation mandated the Carl D. 
Perkins Vocational and Applied Technology Education Act of 1990. 
Recommendations to enhance the impact of prograun improvement and 
evaluation activities are included. The first two sections discuss 
developing new state accountability systems and improving prcjrams 
through annual evaluation, local program improvement plans, and joint 
program improvement plans. The third section addresses ensuring 
increased achievement for special populations through the following 
methods: encouraging efforts to serve special populations; using more 
than the minimum number of measures; evaluating access, learning, and 
labor market outcomes; selecting measures that assess change over 
time; and selecting measures that allow for fair comparisons. A 
discussion of supplementary services necessary to prc^ram improvement 
and eight references complete the document. (KC) 
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THE 1990 PERKINS: EVALUATING AND IMPROVING 
PROGRAM EFFECTIVENESS 

by Maureen Coyle-WiUiams and Carolyn Maddy-Bernstein 

Educational refonn Kii vities during the 1990s are focusing not only on incree «ng the quality 
of education, Inil also cm making jwograms more countable forrcsults. America 2000, Pitsident 
Bush s plan for thieving the National Education Goals, calls for an acccHintability package thai 
will mrcmrage parents, teachers, schools, and ccMnmunitics to "measure lesuits, comjare results 
and insist on change when the results aren't good enough" (II.S. Depanment of Educaliwi, 1991a, 
p. 21). Mandatef within the Carl D. Pierkins Vocational and Applied Technology Edwaticm Act 
of 19W place vocational education at the forefront of the niovement to increase educatic»ial 
|m>gram accountability. If suixessfuU Ae law will improve acactemic and occupational outccmes 
for the growing numtx^r of Americans f^ing unemployment, underemployment, low wages, and 
poverty. 
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Ccmgress expects the 1990 Pferkins Act to generate significant improvements for individuals 
who are members of special pqniiations. Toward this end, vocaticmal educators have been called 
upon to redouble efforts to serve special pqniiations and to focus upon academic as well as 
occupational achievement. InOTdertomc^itc^pix^tresstoward these (^jectives, the 19W Perkins 
legislation mandates the establishment of statewide accountability systems. When implemented, 
these systems will enable local vocational education programs to document improved outcomes. 

Over the next three years, local programs will use measures and standards of performance to 
evaluate im>gress in increasing student achievement, set goals for ne«ssary improvements, and 
implement plans to bring their progiwis up to state defined standards. Widespread improvements 
in vocational programs will be facilitated by the devcloimient of quality systems of accountability 
in each stale. Careful anention must be paid to documenting increased achievement among low- 
achieving students and ensuring that state systems encourage increaj^d access to quality 
vocational programs for individuals who are members of special populations. A firm commit- 
ment to improving outcomes for all students, including those who are members of special 
peculations, is critical This BR/EF describes new requirements for program improvement and 
evaluation in the 1990 Perkins. Recommendations to enhance the imp^ of program improvement 
and evaluation activities are included. 

Developing New State Accountability Systems 

Each state receiving Perkins funds must establish an accountability system by the fall of 1992, 
These systems are to be used lo evaluate student* program, and institution outcomes at the local 
level. Learning, both academic and occupational^ as well as labor market and accessibility 
outcomes are to be examined through the system of measures and standards developed by each 
state. 

While the law mandates multiple objectives for program improvement, requirements for 
statewide accountability systems arc minimal* fttly two measures are required— one measure 
of learning and competency gains (including academic skills learning) and one of the following: 
(a) competency, (b) job or work skill anainn^nt, (c) school retention, or (d) placement. 

Local programs may modify state measures and standards to suit local economic, geographic, 
or demographic factors and/or the char^teristics of the populations to be served. However, Itnal 
modifications must conform to the assessment criteria in the state plan. {Section 1 15 (a)). 
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WHAT ARE PERFORMANCE MEASURES AND STANDARDS? 

(Hoachlamfer* 1990) 

Peiformance measure: The type of cHiicm^ considered a{^ro}^ate for mmivonng. 

Performance standard: The level of perfonnance that individual students, programs, or institutions are 

expected to achieve. 

Performance measures and standards may be defined for students, courses, prognuns, institutions, districts, regions, or 
states. 

Standards may be established to assess performance at a single point in lime {ber«:hmark criteria) and to assess performance 
over time (value added or longitudinal measures). Value added or longitudinal standards require assessing perfonnance 
at different points in time to deierm'mc the amount of learning that has occurred during the period. 



WAYS TO ASSESS GROWTH/PROGRESS 

(Hoachlander, 1990) 



Learning 



Course completion rates 

High school graduation rates 

Rales of completing (tegrees or certificates 

Patterns of course taking — for ei^ample. 
the perceniage of students pursuing a 
planned sequence of vocational and m:a<temic 
courses as cviderxred by student transcripts 



Student achievement on competency*based tests 
Program completion rates 
Attendance rates 

Student achievement on standardized tests 



Labor Market 



Time needed to find and secure employment 
Placement rates 

In a job lelated to naining 

In any job 

In further education or training 
In the military or other service 
(e.g., the Peace Coips) 



Access/Equity 



Entry level wage 

Quarteriy earnings 

Time eniployed in first job 

Rate at which quarterly earnings increase 

Employer and employee satisf^tion 



The percentage of male and female snjdenis and students with special needs enroUed in a specific vocational education 
program compared to the percentage of male and female students and students with special needs in the school population. 

The percentage of students with special needs completing selected vocational education programs compared to the percentage 
of students with special nreds in the school population. 

The percentage of shidents with special needs entering jobs related to training compared to the perceniage of sHKfcnts without 
special needs entering jobs related to training. 



1990 PERKINS REQUIREMENTS FOR STATEWIDE SYSTEMS OF CORE STANDARDS 
AND MEASURES OF PERFORMANCE R)R SECONDARY AND POSTSECONDARY 

VOCATIONAL EDUCATION PROGRAMS 

(Section 115 (b)) 

Systems of perfmnance sfandards and mei^unes must include: 

measum of learning and compHeacy fs^m. Including student progre^ in liasic and mme ^vamred academic 

measures tm one or more id' tl» following: 
compeieiKy anaimmnl, 
job or woii skill attiumnent or enhancen^t, 
school retention or cmiideticm, 

placen^nt into additional training, educaticm, military %rvic«, or emptoymmt; 

incentives or adjustments that: 

encourage seniccs to special populaticms. and 

are develq)ed for each student and are consi^ent with individualiml educ^ticm plans (where appropriate); and 
procedures for using existing resources and methods developed in other federal programs (e^„ JTPA programs). 



Annual Evaluation 



Local Program 
Improvement Plan 
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Improving Programs 

Beginning wiih the 1992-93 school year, local secondary school systems and poslsecondary 
insijtuiions receiving Ftrkins funds musi evaluate the effeciiveness of all vocational-technical 
programs annually. During the evaluation, local programs will a view and evaluate their progress 
as determined by statewide measures and standards of j^rformance (or approved modifications). 
Additionally, program progress in providing xih aUonal srudaiis u iih sirong cxpvriemr in and 
understanding of all aspeas of the industry they are preparing to enter must he evaluated ( Section 
117(a)). Particular attention is to be paid to the status of individuals who are members of special 
populations. 

Pftjgram reviews are to be conducted with the full and informed panicipationof rv»presentaiives 
of individuals who are members of special populations to; 

• identify and adopt strategies to overcome any barriers which are rvsultrng in lower rates of 
access to vocational education programs or success in such programs for individuals who are 
members of special populations; and 

• evaluate the progress of individuals who are members of special populations in % ocaiional 
programs assisted under this act (Section 1 17 (a)). 

The scojxr of local evaluation has been a topic of muchtkbate. Todate.it appears that the U.S. 
Department of Education will require that statewide systems of core performance measures and 
standards be used to evaluate all vocational education programs in local school systems and 
posisecondary institutions regardless of which particular projects, activities, and services are 
fmded under the act (U,S. E^f^nment of Education, proposed regulations, 1991b, Section 
403.191), However, the I^pariment is currently considering allowing the use of sampling or 
other techniques designed to phase in the evaluation requirement and/or increase the flexibility 
allowed in local program evaluation. 

States are responsible for monitoring Perkins-funded programs to ensure adequate progress 
towandl standards (Section 113). However, the annual evaluation process is locally controlled. 
Programs which determine that they have failed to make substantial progress toward established 
standards or identify problems with program access or success for individuals who are members 
of sj^cial populations will be required to develop a local program improvement plan for the 
following school year. The plan must be developed in consultation with teachers, parents, and 
students concerned with or affected by the program, and must describe: 

3 4 



Joint Program 
Improvement Plan 



Encourage Efforts to 
Serve Special 
Populations 



Use More than the 
Minimum Number of 
Measures 



• ho* prog^s will Ik modified to imiTOvetfieir effectively and 

• strategies to improve supplementary services to individuals who are members of special 
pc^ulatiOTs (if necessary KScction 117 (b)). 

Each state is ttsponsiWe for evaluating the success of local progiam improvement plans and 
assisting programs that have not made sufficiem progress toward standards one year after the 
plan's implementation. When this occurs, the state worits ^intly with the necipiem and with 
teachers, parents, and students to develop a joim plan for program improvement. Joint program 
imjnovement plans must COTtain: 

. a description of technical assistance and program activities the State will iMovide to enhance 
local program performance. 

• a reasonable timetable to improve school performance. 

• a description of vocational education strategies designed to improve program performance, 
and 

• a description of --.trategies to improve supplementary services to individuals who are members 
of special populations (if necessary) (Section 1 17 (c)). 

The program improvemem process mandated by the 1990 Perkins is cyclical. There are no 
penalties prescribed by the Kt for prt^rams failing to meet standards even after the implemen- 
tation of a joim program improvement plan. Rather, the law calls for the annual revision of joint 
plans until standards are met for one year. 

Ensuring increased Achievement for Special Populations 

Once in place, state accountability systems could provide the foundation for the program 
improvement efforts described in the law. At the local level, quality accountability systems will 
ensure that programs have objective and credible information on which to base program 
improvement efforts. In addition, such systems will generate infomiation which govemmeiit 
officials, parents, and employers need to determine whether or not a vocational program is 
meeting their needs. However, their effectiveness in stimulating reform will depend upon the 
quality of the systems developed and local commiimem to program improvement. Program 
improvement, as described in the law, must focus on increasing access and improving outcomes 
for individuals from special populations (Section 1 17 (a), 1 17 (b)). Consequently, it is essential 
that state-established measures and si mdards of performance are effective in evaluating and 
encouraging local efforts to serve these groups. Most importantly, state accountability systems 
must not hurt those individuals wham the Perkins is designed to help. 

States and localities must be aware of the imiact which accountability systems have on 
programs serving large populations of individuals with special needs. Rigidly defined standards 
may embarrass and discourage programs serving the highest wmcentrations of students from 
special populations and encourage them to serve only those students who are likely to meet 
st£Uidards. 

Slates wishing to facilitate extensive program improvement will need to go beyond mere 
compliance (Hoachlander. 1991). Director for National Planning and Evaluation at the National 
Center for Research in Vocational Education. E. Gareth Hoachlander (1991), has warned that 
limiting the number of measures used may result in accountability systems which focus on toofew 
outcomes and are of little value in efforts to promote widespread program improvement. He 
arguesthatusefulsystemswilluscbetween6and 12measures. TheU.S.Depaitmemof Educaticm 
has i«commended that States use at least six measures (States Starting, 1 99 1 ). To date, it appears 
that rnost states recognize this need and will include at least five measures in their accountability 
systems (Hoachlantter & Rahn, 1992). 

While willingness to go beyond mere compliance is necessary, it is not enough. Fair and 
accurate systems will require thoughtful and careful selection of performance measures and 
standards. The Center for Uw and Education. Inc. (1991) recommends that accounubility 
systems include measures and standards which: 
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Evaluate Access, 
Learning, and Labor 
Market Outcomes 



Select Measures that 
Assess Change Over 
Time 



^lect Mrasures That 
Allow for Fair 
Comparisons 



• assess stiKlent achtevcmcm of basic and advanced academic skills, including problem solving 
in roaih. science, reading* waiting, and social studies; 

• asscssprogramprovisicMiof strong experience and understanding of all aspects of ihe industry; 

• examine the full range of relevMit program features; 

• use ob^tive, reliable, and accurate data; and 

• asKss student skills with measures that are valid for all populaucwis. 

Local {HOgress in increasing jmigram ^ess for individuals who aie members of special 
peculations may be monitcHrd through selected measures. However, access measures alone are 
not sufficient for monitoring the progress of spwial p<v"lations. They must be used in 
conjunction with learning and labw market measures. While it is not necessary or desirable to 
establish separate measures or standards for special populations, it is critical that state systems 
include measures that identify and encourage achievement gains for all groups. Perfomfiance 
measuies which docun^nt die progress of low-a:hieving students must be included within each 
state's system. Hoachlamter (1991) recommends that states and localities adopt performance 
me^ures that reflet studc^it gains over time. 

Students wIk) enter a program with very low skills may achieve significant 
gains in skills, even though their skill levels may still fall short of dte 
performance standard required for program completion. And such gains 
should be identified and progress encouraged, (p. 21) 

In order to encourage local effons to serve special population students, accountability systems 
must report and compare performance fairly. 

Comparing raw scotcs of markedly different students, programs or institutions 
is not likely to result in an accurate picture of relative performance. An inner 
city school with high concentrations of academically disadvantaged students 
is not likely to score as well on a wide variety of performance measures as a 
school in a middle class suburb, A direct ccnnparison of the performance of 
these two schools is clearly not appropriate. (Hoachlander* 1991, p- 65) 



Supplementary Services are Necessary to Program Improvement 

The 1990 Pericin^s mission is to **make the United States more competitive in the world 
economy by developing mrae fully the academic and occupaiiOTal skills of all segments of the 
population*' {P.L 101-392, Section 2). This mission cannot be achieved without significant 
improvements in services to individuals who are nwimbers of special populations. TTie law 
acknowledges the importance of imptoving outcomes for these populations and includes 
numerous provisions designed to accomplish widespread increases in academic and occupational 
achievement. The program improvement process mandated in the law requires: (1) ongoing 
assessment of the progress of special populations, (2) identificatic^ of strategies to overcome 
barriers to pnogram access and success for these individuals, and (3) the identification and 
improvement of strategies to improve supplementary services to individuals who are members of 
special populations. 

The provision of supplementary services is as critical to program improvement as is the 
development of quality accountability systems. To a great extent, generating significant 
improvements in populations of individuals with special needs will depend upon the supplemen- 
tary seivices called for in the 1990 I^rkins Aa. However, resistance to the law's mandate to 
improve the provision of supplementary services may severely compromise its impact on those 
groups targeted in the law (i.e. individuals with disabilities, those who are disadvantaged either 
educationally or economically, persons with limited proficiency in English, individuals in 
correctional institutions, and those participating in programs designed to eliminate sex bias 
{Section 521 (31)). 
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The U.S. Dq>artinem of Educaiicm is cur^ienUy considering changes in the 19% 
Pierkins* xequiiements for services to special pc^ulaticms in respcm^ to critics wiw 
argue that they limit the fiinds which can be spent on program improvement. For a 
signifkram nun^r of ywih in this coumry, siq^plementary senices are the mos! 
fiecessary program improvprnenL Wifhom them, they wiil fail. States and localities 
that decick to limit suf^iementary »rvices in favor of ^'program improvemert"' risk 
substandard outccmies for a substantial number of their students. 
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